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Summary 


The  present  study  reviewed  some  of  the  coranonly  used  indices  of  interrater  agreement  and  determined  advantages  and 
disadvantages  of  each  when  used  to  assess  the  reliability  of  quantitative  job  analysis  scores  derived  from  detailed  nar- 
rative  job  descriptions.  A  listing  was  obtained  of  all  civilian  employee  positions  and  job  titles  from  a  medium-sized 
military  hospital  employing  approximately  1,100  persons,  This  listing  included  89  separate  job  categories,  each  uniquely 
identified  by  civil  service  commission  job  code,  pay  grade  range,  and  a  designator  noting  whether  or  not  the  position 
was  supervisory  in  nature.  Extensive  narrative  descriptions  of  each  of  the  89  job  categories  were  obtained  from  the 
U.S,  Civil  Service  Commission  Qualifications  Standards  (1978).  These  descriptions  contained  detailed  information  about 
the  scope  of  job  duties,  experience  and  training  requirements,  supervisory  controls,  and  general  work  conditions  typically 
encountered  by  incumbents  in  each  job  category. 

Four  graduate  level  psychology  students  trained  in  the  use  of  the  Position  Analysis  Questionnaire  (PAQ)  were  asked 
to  rate  the  above  job  descriptions.  The  PAQ  is  an  extensive  questionnaire  divided  into  six  major  sections  (Information 
Input,  Mental  Processes,  Work  Output,  Relationships  with  Other  People,  Job  Context,  and  Other  Job  Characteristics).  Job 
analysts  are  required  to  evaluate  different  aspects  of  jobs  in  each  section  and  to  record  their  evaluations  of  varicu* 
job  attributes  on  either  five-point  Likert  type  or  c’.ichotomous  (0-1)  scales.  Twenty-five  jobs  were  randomly  selected 
for  rating  by  all  four  raters;  the  remaining  64  job  descriptions  were  rated  by  two  raters.  Each  of  the  ices  of  inter¬ 
rater  agreement  reviewed  (i.e,,  percent  agreement,  Kappa,  Weighted  Kappa,  Porrson  product-moment  correlation,  intraclass 
correlation,  and  the  Spearman-Brown  formula)  was  computed  for  comparative  purposes.  The  results  indicated  general  agree¬ 
ment  or.  the  ratings  obtained,  but  differences  were  noted  in  the  estimates  produced  by  ^the  various  indices.  Reasons  for 
such  differences  were  explored  and  recommendations  made  for  avoiding  potential  difficulties  in  assessing  interrater 
agreement  on  job  analysis  ratings. 
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Assessing  Interrater  Agreement  in 
Job  Analysis  Ratings 

Research  interest  in  job  analysis  and  job  classification  is,  once  again,  on  the  rise  (Cornelius,  Carron,  &  Collins, 
1979;  Tornow  &  Pinto,  1976).  As  detailed  by  Jones,  Main,  Butler,  and  Johnson  (Note  l),  this  resurgence  of  interest  is 
stimulated  in  part  by  the  need  to  obtain  precise,  objective  data  for  use  in  job  development,  performance  appraisal,  and 
other  such  situations  in  an  efficient  and  cost-effective  manner.  Traditionally,  job  analysis  information  has  been  ac¬ 
quired  by  having  expert  observers  rate  the  behaviors  of  job  incumbents  or  by  having  incumbents  complete  lengthy  question¬ 
naires  such  as  the  Position  Analysis  Questionnaire  (McCormick,  Jeanneret,  &  Mecham,  1972).  Issues  related  to  time,  cost, 
intrusiveneoa ,  and  reliability  of  the  information  obtained  by  either  procedure,  however,  have  led  to  the  search  for  al¬ 
ternative  methods  for  obtaining  job  analysis  data. 

One  alternative  method  capitalizes  on  the  fact  that  many  organizations  have  conducted  extensive  job  analyses,  but 
have  reduced  such  data  to  detailed  narrative  descriptions  about  either  individual  job  requirements  or  requirements  for 
job  families.  Although  such  narrative  descriptions  arc  rich  with  information,  they  lack  the  numerical  precision  nec¬ 
essary  for  most  research  and  applied  uses.  While  it  is  possible  to  again  conduct  the  original  job  analyses  and  thus 
produce  more  quantitatively  oriented  information,  a  more  efficient  technique  would  be  the  development  of  numerical  ratings 
based  on  existing  job  descriptions  (cf.  Jones,  et  al.,  Note  1). 

The  utility  of  such  a  method  requires  at  a  minimum  that  trained  raters  agree  on  the  ratings  to  be  assigned  to  each 
position  (i.e.,  one  must  address  interrater  reliability).  However,  a  critical  issue  in  detexmining  agreement  is  the 
selection  of  a  statistical  method  appropriate  to  the  data  collected  (Jones  &  James,  1979).  Because  of  the  many  indices 
of  rater  agreement  in  common  use  today,  the  problem  of  selecting  an  appropriate  index  seems  particularly  difficult.  The 
purpose  of  the  present  study  is  to  review  some  of  the  conmonly  used  indices  of  interrater  agreement  and  to  determine  ad¬ 
vantages  and  disadvantages  of  each  when  used  to  assess  agreement  in  job  analysis  ratings  derived  from  narrative  descrip¬ 
tions  . 

Indices  of  Interrater  Agreement 

Percent  agreement  is  a  commonly  used  index  but  has  a  number  of  problems  associated  with  it  (Cohen,  1960;  Cohen, 

1968;  Jenkins,  Nadler,  Lawler,  &  Cammann,  1975;  Mitchell,  1979;  Repp,  Deitz,  Boles,  Deitz,  &  Repp,  1976).  First,  this 
statistic  includes  chance  agreement  and  may  overestimate  tru  agreement  among  observers.  Paradoxically,  percent  agree¬ 
ment  may  also  underestimate  interrater  reliability  on  rating  scales  requiring  complex  decisions  because  it  fails  to  con¬ 
sider  partial  agreement. 

To  compensate  for  the  shortcomings  of  percent  agreement,  Cohen  (1968)  introduced  Kappa  (K)  and  Weighted  Kappa  (Koo) . 
These  statistics  assess  agreement  after  correcting  for  chance,  while  Ku>  allows  partial  credit  to  ratings  which  are  similar 
but  not  identical.  On  the  surface,  K  seems  to  resolve  the  problems  faced  by  percent  agreement.  However,  K  assumes  that 
the  selection  of  a  category  in  one  observation  is  independent  of  the  selection  of  a  category  on  successive  observations. 
Further,  observations  should  not  be  concentrated  in  a  few  cells.  Such  assumptions  may  be  easily  violated  if  one  is  con¬ 
ducting  job  analyses  on  similar  jobs  that  are  relatively  few  in  number.  Assume  that  two  individuals  are  rating  a  homo¬ 
genous  set  of  jobs  or  a  single  five-point  scale.  Further,  assume  that  the  attribute  is  obvious  ami  easy  to  rate.  The 
homogeneity  of  the  rated  group  increases  the  probability  that  one  may  find  a  high  proportion  of  scores  occurring  in  one 
or  two  cells  while  other  cells  have  small  or  nonexistent  proportions.  This  example  violates  the  assumptions  upon  which 
K  is  based  and  will  tend  to  reduce  th^  power  of  the  test  to  a  point  of  providing  little  useful  information  (Overall, 

1980) . 
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The  Pearson  product -moment  correlation  coefficient  (r)  hat  also  been  used  by  a  number  of  authors  as  an  index  of 
interrater  agreement  (Bernardin,  Alvares,  6  Cranny,  1976;  Cornelius,  et  al.,  \979;  Jenkins,  ot  al.t  1975;  Taylor  6  Colbert, 
1978).  However,  Ebel  (1951)  noted  that  only  pairwire  comparisons  can  be  made,  the  table  of  ratings  must  be  complete, 
and  the  betveen-rater  variance  is  always  removed  in  calculating  the  product -moment  formula.  Unfortunately,  Ebel  pointed 
out  situations  where  between-rator  variance  should  be  included  as  part  of  the  error  term.  Namely,  when  comparisons  are 
made  among  single  raw  acorea  assigned  to  different  subjects  by  different  raters,  the  between-rater  variance  should  be 
included  in  the  error  term.  For  these  reasons,  Ebel  (1951)  and  others  (Bartko,  1966,  1976;  Selvage,  1976;  Winer,  1971) 
advocated  the  use  of  the  intraclass  correlation  as  an  index  of  interrater  agreement  because  it  permits  the  researcher  to 
choose  whether  or  not  to  include  the  between-rater  variance. 

Despite  this  advantage,  the  intraclass  correlation  (because  of  its  basis  in  the  ANOVA  paradigm)  suffers  many  of  the 
same  criticisms  mentioned  earlier  for  K.  For  example,  Selvage  (1976)  noted  that  ANOVA  fails  to  produce  coefficients  which 
reflect  the  consistency  of  ratings  when  assumptions  of  normality  are  grossly  violated.  Further,  ANOVA  requires  substan¬ 
tial  between-item  variance  to  produce  a  significant  indication  of  agreement  (Finn,  1970;  Selvage,  1976).  Thus  an  example 
similar  to  the  one  given  earlier  for  K  would  also  violate  the  underlying  assumptions  of  ANOVA  and  distort  the  intraclass 
correlation  as  an  index  of  interrater  agreement. 

Lastly,  Jones  and  James  (1979)  noted  that  indices  of  reliability  of  mean  scores  among  raters  (e.g.,  the  Spearman- 
Brown  formula)  have  also  been  used  in  assessing  interrater  agreement.  They  argued,  however,  that  "the  Spearman-Brovn 
formula  is  not  well  suited  to  large  numbers  of  raters.  ...  large  sample  sixes  tend  to  yield  high  estimates  of  mean  score 
reliability  even  when  relatively  heterogeneous  individual  scores  are  used  to  compute  such  means.  Thus,  the  reliability 
of  the  mean  scores  appears  to  provide  an  overly  optimistic  estimate  of  agreement."  (1979,  p.  207).  Jones  and  James, 
after  reviewing  many  of  the  same  indices  discussed  above,  concluded  that  no  single  index  was  adequate  for  all  situations. 
They  suggested  that  more  than  one  index  of  interrater  agreement  may  be  necessary  to  assess  reliability  among  raters. 

On  the  basis  of  such  recommendations  and  because  the  weaknesses  of  some  of  the  indices  appear  to  be  offset  in  other 
indices  of  agreement  (and  vice  versa),  we  computed  several  of  the  available  indices  to  determine  whether  the  combination 
of  trained  raters  and  structured  job  analysis  questionnaires  could  produce  a  viable  method  for  translating  the  d* tailed 
narrative  descriptions  of  jobs  present  in  many  organizations  to  quantified  data  useful  in  performance  assessment,  job 
classification,  and  so  forth. 

Method 

A  listing  was  obtained  of  all  civilian  employee  positions  and  job  titles  from  a  medium-sized  military  hospital  em¬ 
ploying  approximately  1,100  persons.  This  listing  included  89  separate  job  categories,  each  uniquely  ident:fied  by  civil 
service  commission  job  code,  pay  grade  range.,  and  a  designator  noting  vhe^ner  or  not  the  position  was  supervisory  in 
nature.  Extensive  narrative  descriptions  of  each  of  the  89  job  categories  were  obtained  from  the  U.S.  Civil  Service 
Comoisaion  Qualifications  Standards  (1978).  These  descriptions  contained  uetailed  information  about  tne  scope  of  job 
duties,  experience  and  training  requirements,  supervise cy  controls,  and  general  work  conditions  typical l'*  encountered  by 
incumbents  in  each  job  category. 

Four  graduate  level  psychology  students  trained  in  the  use  of  the  Position  Analysis  Questionnaire  (PAQ)  were  asked 
to  rate  the  above  job  descriptions.  The  PAQ  is  an  extensive  questionnaire  divided  into  six  major  sections  (Information 
Input,  Mental  Processes,  Work  Output,  Relationships  with  Other  People,  Job  Context,  and  Other  Job  Characteristics).  Job 
analysts  are  required  to  evaluate  different  aspects  of  jobs  in  each  section  and  to  record  their  evaluations  of  various 
job  attributes  on  either  five-point  Likert  type  or  dichotomous  (0-1)  scales.  Twenty-five  jobs  were  randomly  selected 
for  rating  by  all  four  raters;  the  remaining  64  job  descriptions  were  rated  by  two  raters.  Each  of  the  indices  of  inter¬ 
rater  agreement  discussed  earlier  was  computed  for  comparative  purposes. 
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Results 


Table  1  presents  «  summary  of  the  reliability  estimates  obtained  on  the  PAQ  ratings  using  each  technique.  These 
estimates  are  presented  in  terms  of  the  number  of  coefficients  lying  within  a  given  range.  In  general  K  and  Kaj  provided 
the  most  conservative  estimates  of  interrater  agreement,  producing  median  values  of  .19  and  ,22,  respectively,  Con¬ 
versely,  percent  agreement  and  Spearman-Brown  approaches  provided  the  mosi  optimistic  indications  of  interrater  agreement, 
with  median  values  of  .63  and  .73,  respectively.  Finally,  the  average  pairwise  correlation  and  intraclass  techniques 
occupied  an  intermediate  position  as  indicated  by  median  values  of  .31  and  .39. 

Table  1 


FREQUENCY  DISTRIBUTION  OF  ESTIMATES  OF  INTERRATER  AGREEMENT  ON  THE  PAQ 

Weighted  Percent  Average  Intraclass  Spearman 

Kappa  Kappa  Agreement  Correlation  Correlation  Brown 


Range 

Freq. 

Cum  X 

Freq . 

Cum  X 

Freq . 

Cum  X 

Freq . 

('urn  X 

Freq 

.  Cum  X 

Freq. 

Cura  X 

.91  - 

1.00 

i 

0.7 

i 

0.7 

8 

5.9 

2 

1.5 

i 

0.7 

12 

8.8 

.81  - 

.90 

- 

0.7 

- 

0.7 

18 

19.1 

5 

5.1 

5 

4.4 

27 

28.7 

.71  - 

.80 

- 

0.7 

- 

0.7 

26 

38.2 

17 

17.6 

6 

8.8 

31 

51.5 

.61  - 

.70 

- 

0.7 

4 

3.7 

25 

56.6 

18 

30.9 

’2 

17.6 

18 

64.7 

.51  - 

.60 

5 

4.4 

5 

7.3 

13 

66.2 

27 

50.7 

16 

29.4 

3 

66.9 

.61  - 

.50 

10 

11.8 

17 

19.8 

21 

81.6 

20 

65.4 

24 

47.1 

3 

72.8 

.31  - 

.40 

14 

22.0 

22 

36.0 

12 

90.4 

9 

72.1 

22 

63.2 

7 

77.9 

.71  - 

.30 

35 

47.8 

23 

53.0 

10 

97.8 

11 

80.1 

6 

67.6 

11 

86.0 

.11  - 

.20 

35 

73.5 

29 

74.3 

3 

100  0 

13 

89.7 

15 

78.7 

2 

87.5 

< 

.10 

36 

100.0 

35 

100.0 

- 

100.0 

14 

100.0 

29 

100.0 

17 

100.0 

Mud  ian 

Value 

.19 

.22 

.63 

.50 

.39 

.73 

While  the  above  results  describe  general  trends,  an  examination  of  estimates  obtained  for  each  item  suggested  addi¬ 
tional  trends  of  interest.  As  shown  in  Table  2,  some  of  the  indices  appeared  to  be  more  sensitive  to  characteristics  of 
the  underlying  distribution  than  others.  When  there  wa6  high  agreement  aiming  raters  ano  limited  variance  on  the  char¬ 
acteristic  being  rated  (Condition  A),  all  of  the  indices  except  percent  agreement  tended  to  produce  unrealistically  low 
estimates  of  agreement.  With  greater  variance  or  the  rated  item,  there  was  considerably  more  consistency  in  the  esti¬ 
mates  obtained  in  cither  favorable  (high  agreement  among  taters;  Condition  B)  or  unfavorable  (lew  agreement  among  raters; 
Condition  C)  directions. 
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Table  1 


Item  Level  Comparisons  Between  Intimate*  of  Interrater  Agreement  and  Underlying  Distributional  Characteristics 

INDEX 


Distr ibutior.al  Characteristic 

Condi t ions 

Kappa 

Weighted 

Kappa 

Percent 

Agreement 

Average 

Correlation 

Intraclass 

Correlation 

Spearman* 

Brown 

A.  Homogeneity  across  jobs  and  ^ 
high  agreement  among  raters. 

.04 

.04 

.77 

-.01 

o 

w 

-.14 

a ,  Hot  o  r  o  g  u  n  o  i  tv  a  c  r  o  s  s  j  ob  s  aO\J 
high  a g r c c me n t  among  raters." 

.43 

.43 

.83 

.79 

.74 

.92 

C.  Heterogeneity  across  jobs  a’^d 
low  agreement  among  raters. 

.01 

.04 

.16 

.  .13 

.05 

-.23 

Calculations  based  on  ratings  of  PAQ  item  number  2?,  “Body  balance  (sensing  the  position  and  balance  of  the  body 
when  body  balance  is  cr it ical  to  job  performance,  as  when  walking  on  beams,  climbing  high  poles,  working  on  steep 
roots,  walking  on  slippery  floors,  etc.),  rated  in  terms  of  "importance  to  this  job." 

o 

"Calculations  based  on  ratings  of  VAC  item  number  21,  "Far  visual  differentiation  (seeing  differences  in  the  details 
of  objects,  events,  or  features  b  evond  o  rm *  a  reach,  for  example,  operating  a  vehicle,  landscaping,  sports  officiat¬ 
ing,  etc,),  rated  in  terms  of  "inport.n.vo  to  this  job." 

^Calculations  based  on  racings  of  PAQ  item  number  153,  "Non-j  b  required  social  contact  (the  opportunity  to  engage 
in  informal ,  w o n -job  r o g u i rod  conversation,  social  interaction,  etc.  with  other  while  on  the  job,  for  example,  barber, 
taxi  driver,  receptionist,  journeyman  cr.d  apprentice,  ote.j  do  not  include  personal  contacts  r cnuirc-d  by  the  job), 
rated  in  terms  of  "frequency  of  opportunity." 
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The  foregoing  results  indicate  that  at  an  item  level,  only  under  one  condition  (Condition  B  in  Table  2)  did  inter¬ 
rater  agreement  approach  acceptable  levels.  Thus,  reliability  estimates  wore  also  calculated  for  dimension  scores  (see 
McCormick,  et  al,,  1972).  These  estimates,  shown  in  Table  3  in  terms  of  the  number  of  coefficients  lying  within  a  given 
range,  were  generally  higher  than  those  obtained  for  items  and  appeared  to  represent  acceptable  levels  of  agreement. 

Discussion 

The  present  study  explored  the  reliability  of  job  analysis  scores  based  on  narrative  job  descriptions.  The  computa¬ 
tion  of  several  of  the  commonly  used  indices,  however,  posed  difficulties  for  drawing  straightforward  conclusions  about 
interrater  agreement.  First,  some  indices  tended  to  provide  generally  higher  or  lower  estimates  than  others.  For  ex¬ 
ample,  the  median  estimates  provided  by  percent  agreement  and  the  Spearman-Brown  correction  were  more  than  three  times 
the  magnitude  of  those  provided  K  and  Kao.  Second  the  relative  differences  among  the  indices  varied  according  to  the 
nature  of  the  distribution. 

All  indices  appeared  sensitive  to  situations  where  there  was  little  or  no  agreement  among  the  raters,  yielding  re¬ 
liability  estimates  that  were  low  and  similar  in  magnitude.  Likewise,  when  agreement  was  high  and  there  was  heterogeneity 
on  the  characteristics  being  ratod,  all  indices  yielded  relatively  high  reliability  coefficients.  Kvon  in  these  cases, 
however,  K  and  weighted  Kw  provided  generally  lower  estimates  than  the  others.  The  greatest  difficulty  came  in  situations 
where  there  was  substantial  homogeneity  on  the  characteristic  and  the  raters  wore  able  to  reach  high  agreement.  In  this 
condition,  percent  agreement  tended  to  reflect  t  »  actual  level  of  interrater  agreement  whereas  the  other  estimates 
tended  to  be  quite  low. 

These  findings  suggest  that  the  user  must  exerciae  caution  in  selecting  and  interpreting  indices  of  interrater 
agreement.  When  there  is  heterogeneity  among  jobs  on  the  characteristics  being  rated,  the  user  need  be  sensitive  onlv 
to  the  degree  to  which  a  particular  index  is  likely  to  be  systematically  higher  or  lover  than  another.  However,  when 
there  is  reason  to  suspect  that  the  jobs  being  rated  will  be  quite  similar,  some  of  the  estimates  may  lead  to  erroneous 
conclusions  that  rater  agreement  is  low.  Thus,  th'  user  must  consider  the  underlying  distribution  of  the  ratings  in 
reaching  conclusions  about  rater  agreement. 

Bearing  such  points  in  mind,  it  appears  that  the  raters  were  able  to  reach  at  least  moderate  levels  of  agreement  in 
developing  scores  from  narrative  descriptions  of  complex  jobs.  For  the  dimension  scores  at  least,  the  estimates  appear 
consistent  with  indices  reported  in  the  literature.  For  example.  Smith  and  Hakel  (1979)  reported  mean  reliabilities  for 
the  PAQ  (baaed  on  pairwise  correlations)  ranging  from  ,49  to  .63.  Taylor  and  Colbert  (1978)  use'1,  a  similar  technique 
and  reported  an  average  correlation  of  .68  among  pairs  of  raters  while  McCormick,  et  al.,  (1972)  reported  mean  reliabili¬ 
ties  of  .80.  Based  on  these  values,  the  reliabilities  of  the  present  effort  appeared  at  least  acceptable. 

It  should  be  noted,  however,  that  many  different  and  often  interrelated  factors  exist  which  can  affect  or  distort 
any  index  of  reliability.  In  the  present  context,  for  example,  differences  in  understanding  of  the  rating  format  and 
instructions  provided  by  the  PAQ  or  difference  in  the  education,  experience,  and  other  such  background  characteristics 
between  raters  are  plausible  sources  of  confounding  in  obtaining  identical  information  from  narrative  job  descriptions. 

To  avoid,  or  at  least  minimise,  problems  of  this  type  Ka)*(1980)  noted  the  importance  of  establishing  the  reliability 
of  data  (regardless  of  type)  at  the  same  level  that  it  will  be  used  to  address  research  questions.  Our  results  suggest 
that  trained  raters  are  able  to  use  narrative  job  descriptions  to  develop  quantitative  job  profiles,  but  only  at  the  job 
dimension  or  job  characteristic  composite  level.  Thus,  such  ratings  likely  to  differentiate  primarily  among  types  of 
jobs.  The  poor  reliability  associated  with  item  level  analysis,  coupled  with  the  fact  that  the  PAQ  is  based  on  generic 
terminology  and  lacks  job  specific  language  (Levine,  Ash,  6  Bennett,  1980)  would  tend  to  obscure  the  subtle  differences 
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that  arc  necessary  to  distinguish  among  jobs  of  the  same  type. 

The  degree  to  which  the  loas  of  such  distinctions  la  problematic  depends  upon  the  uae  to  which  the  job  analysis 
is  put.  Many  of  the  common  uses  of  such  data  are,  in  fact,  more  consistent  with  a  job  family  approach  than  with  an 
ipsative  job  approach.  For  example,  Levine,  et  al.,  (1980)  compared  the  utility  of  different  job  analysis  techniques 
for  the  development  of  selection  test  batteries.  They  concluded  that  M...  no  matter  how  rich  in  detail  a  job  analysis 
report  may  be,  reaultant  exam  plana  will  not  vary  enough  to  produce  significantly  different  evaluations  of  their  quality,11 
(p.  534),  Similar  conclusions  are  likely  for  situations  where  job  analysis  is  used  for  setting  salary  levels,  or  de¬ 
signing  training  programs. 

Insofar  as  many  of  the  applied  and  research  uses  of  the  PAQ  arc  concerned  with  the  classification  of  jobs  into 
similar  families,  the  uae  of  narrative  job  descriptions  to  derive  quantitative  scores  about  such  families  provides  a 
relatively  quick,  inexpensive,  and  unobtrusive  means  of  obtaining  needed  data.  Thus,  the  technique  appears  to  overcome 
many  of  the  objections  to  fc.he  current  reliance  on  job  incumbents  or  trained  observer  for  settings  where  detailed  infor¬ 
mation  about  relevant  jobs  already  exists. 
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